Recently, products such as TVs having a display which uses super-resolution technology are marketed. The super-resolution technology reproduces detailed portions (components of not less than a Nyquist frequency) lost at the time of sampling pixels of an image (at the time of photoelectric conversion by an image pickup device and reduction processing of an image) by image processing. For example, when super-resolution technology is used for displaying a video of a video camera of the NTSC system and a video of a DVD with low resolution on a large-screen liquid crystal TV, the detailed portions are reproduced and thereby a clear image with high resolution can be obtained.
Various systems have been proposed for super-resolution technologies. They are a super-resolution in a frame system, a dictionary based super-resolution system, and a multi-frame super-resolution system and so on. A multi-frame super-resolution system of them has particularly a high reproduction factor of detailed portions of an image, and has high performance. The multi-frame super-resolution system performs an alignment processing and a reconfiguration processing. The alignment processing, from a plurality of frames which are backward and forward in terms of time (a standard frame and a reference frame), detects the same portions of the same object as the corresponding points. The reconfiguration processing generates an image with high resolution using the information of the corresponding points of the reference frame and the standard frame obtained by the alignment processing. Thus a high resolution frame of one frame is generated from an amount of information larger than an amount of information of a single frame. The alignment processing and the reconfiguration processing are continuously executed while the frames are brought forward one by one, and thereby a video with high resolution can be generated. In addition, the reconfiguration processing includes a system using a repetition processing in which the image quality is high but the processing load is also high, and a system without using a repetition processing in which the image quality is low but the processing load is also low.
By the way, a multi-frame super-resolution processing includes two embodiments of a software processing by use of a CPU, and a dedicated hardware processing. Conventionally, it prevailed that an arithmetic processing unit is mounted in a personal computer. In a software processing by a CPU in the personal computer like this, each processing of a multi-frame super-resolution processing is executed in sequence. That is, an alignment processing of a precedent frame with a standard frame is executed, and an alignment processing of the standard frame with a subsequent frame is executed, and finally, a reconfiguration processing is executed. Or, there may be a case in which pseudo parallel processings are executed with a software programming technology named multithread, in a time-sharing mode and in order of priority which an OS (Operating system) controls. But, in this technology, there may be even a case which requires more time by the overhead for switching over the processings than a case in which the processings are executed sequentially.
Recently, high performance products each having a CPU with a plurality of internal arithmetic units have prevailed. An arithmetic processing unit is called a core, and having a plurality of cores, such a CPU is called a multi-core CPU or a many-core CPU. The CPU can operate a plurality of programs in parallel using a plurality of the arithmetic processing units.
In addition, not limited to a CPU which is mounted on a personal computer, it has been progressing that an embedded CPU and a DSP (Digital Signal Processor) are formed to have multi cores or many cores. If a plurality of arithmetic processing units is utilized, an alignment processing of a precedent frame with a standard frame, an alignment processing of the standard frame and a subsequent frame, and a reconfiguration processing can be executed in parallel.
However, in the case provided with single arithmetic processing unit, it is necessary to align all pixels in the precedent and subsequent reference frames with the standard frame which is to be enlarged in the super-resolution processing, and since a repetition search (a block matching) is performed within a frame for a pattern similar to a block with a definite size, it was a problem that the processing load is high. In addition, even in the case in which a system using a repetition processing is employed as the reconfiguration processing so as to obtain high image quality, there was a problem that the processing load becomes high. Even if a multi-core CPU is used to address these problems, in order to operate a program in parallel using a plurality of arithmetic processing units, ingenuity in an algorithm is required, and it is not possible at present to automatically operate a program in parallel. Consequently, an algorithm for multi-frame super-resolution has been desired which can effectively operate a plurality of arithmetic processing units of a multi-core CPU in parallel.
On the other hand, memory architecture of recent years has a two-hierarchy configuration (more hierarchies, sometimes) composed of a main memory and a cache memory. A main memory has a large capacity, but the speed thereof drops at the time of random access. Though a cache memory has a small capacity, it operates at a high speed at the time of random access. In a software processing by a CPU, and also in a dedicated hardware processing, there is a problem of a memory access band. Unless the data required for processing is preferably arranged on a cache memory, the processing speed might drop by a bottle neck of a memory access band between the main memory and the cache memory. Since the image data has a large size in particular, and these problems come to the front, solutions to the drop in the processing speed have been desired.